3D Convolutional Neural Network for Radiation Toxicity Prediction
Back in 2023, I was working on a knowledge‑based planning initiative for head and neck radiation treatments. Around that same time, commercial auto‑contouring tools were rapidly emerging, and the broader radiation oncology community was beginning to embrace machine learning in a meaningful way. That convergence pulled me deeper into neural networks—not just as a curiosity, but as a potential extension of the physics skillset.
One of the persistent challenges in head and neck radiotherapy is managing late toxicities, particularly salivary gland dysfunction. Xerostomia remains one of the most common and debilitating side effects, and in more severe cases, patients progress to dysphagia, significantly impacting quality of life. I began wondering whether we could leverage modern deep learning techniques to predict these toxicities before treatment—using the same imaging and dose information we already generate in routine clinical workflows.
That idea eventually led to a collaboration with colleagues at Duke University, where we developed a 3D convolutional neural network designed to predict grade ≥2 xerostomia at six months post‑treatment.
ASTRO 2024 Poster
Building the IDS‑Based 3D CNN
Our approach centered on constructing a unified 3D image representation that captured the essential components of head and neck radiotherapy. We created what we called the IDS image—a three‑channel volumetric input consisting of:
Intensity: the planning CT
Dose: the 3D dose distribution
Structure: a labeled mask of relevant OARs
This allowed the network to learn spatial relationships between anatomy, delivered dose, and tissue susceptibility in a single coherent tensor.
The model architecture used four convolutional layers with ReLU activation, interleaved with pooling operations to progressively extract hierarchical features. After the convolutional backbone, we incorporated a multilayer perceptron (MLP) that merged the learned imaging features with clinical parameters such as age, sex, weight, follow-up status, and treatment characteristics. The goal was to combine radiobiological context with patient‑specific factors to produce a more holistic prediction.
Why Neural Networks Matter in Radiation Oncology
Although this project was focused on xerostomia, the broader implications are what matter most. Radiation oncology generates enormous volumes of structured and semi‑structured data—CTs, dose grids, contours, plan metrics, toxicity outcomes, and longitudinal follow‑up. Historically, much of this information has been siloed within individual institutions, limiting our ability to build robust predictive models.
Neural networks thrive on scale, diversity, and multimodal inputs. When applied thoughtfully, they can help us:
anticipate toxicity before treatment begins
identify patients who may benefit from adaptive strategies
improve plan quality through automated feedback
support clinical decision‑making with data‑driven insights
Encouragingly, the field is beginning to move toward more open data ecosystems. Public repositories such as The Cancer Imaging Archive (TCIA) and the Medical Imaging and Data Resource Center (MIDRC) already host large, well‑curated imaging datasets that demonstrate how powerful shared resources can be.
These platforms provide a glimpse of what becomes possible when imaging, annotations, and clinical metadata are made accessible in standardized formats. If similar large‑scale treatment datasets—complete with dose distributions, contours, and longitudinal outcomes—were available through secure, multi‑institutional frameworks, the field could accelerate model development dramatically. We could train networks not only to predict xerostomia, but also dysphagia, fibrosis, feeding‑tube dependence, local control, and even survival, using the same planning information we rely on every day.
